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The Relative Effectiveness Of Structured Questions And Summarizing On 

Near And Far Transfer Tasks 

Weimin Wang 

North Carolina Wesleyan College 

Abstract 

T/iis study is to compare the effect of two learning strategies: summarking and structured questions on near and far transfer 
tasks. The hypothesis is that the structured questions will better activate metacognitive and critical thinking skills than 
summarizing so that it will better facilitate learning transfer . However, the results do not show any significant difference between 
two strategies on transfer tasks. Possible reasons that the students failed to activate metacognitive skills or critical thinking skills 
are discussed. Recommendations to maximize the effect of structured questions on transfer tasks are also discussed. 



Introduction 

Various studies (Sternberg & Frensch, 1993; Salomon & Perkins, 1989; Perkins & Grotzer, 1997; Kosonen & Winne, 1995- 
Stolovitch & Yapi, 1997) have been conducted in the field of educational research in general, and instructional systems design in 
particular, on how to facilitate transfer of learning. The early research on learning transfer focused on the role of practice. Most 
early researchers agreed that increased amounts of practice would increase transfer (Thorndike, 1913; Gagne & Foster, 1949; 
Hovland, 1951). More recently, one major interest is the role of metacognitive strategies and critical thinking in learning transfer. 
Most of the research focused on teaching metacognitive strategies and critical thinking skills. However, relatively little research 
has focused on the activation of metacognitive strategies and critical thinking skills that ultimately promotes learning transfer 
(Forster, 1996). 

This study explores the possible way to activate metacognitive strategies and critical thinking skills through the use of 
reflective activities, like summarizing or answering structured questions after reading. Metacognitive strategies refer to the 
regulation and control of cognition. Despite differences, most researchers would agree critical thinking involves the intentional 
application of rational, higher order thinking skills that include analysis, synthesis, problem recognition and problem solving 
inference and evaluation (Angelo, 1995). Structured questions in this study refer to a set of generic questions that are not’ 
associated with specific content of any articles. Also, this research studies the effect of metacognitive and critical thinking skill 
activation on near and far transfer. Near transfer is the application of learning to situations similar to those in which initial 
learning has taken place. In contrast, far transfer is the application of learning to situations dissimilar to those of the original 
learning events. & 

The purpose of this study was to compare the relative effectiveness of structured questions and summarizing on near and far 
transfer tasks. More specifically, this study was designed to address the following questions: 

1. Will students who answer specific structured questions perform better on near 

transfer task than students who are simply asked to summarize the content? 

2. Will students who answer structured questions perform better on far transfer 

task than those who only write a summary of an article? 

It was expected that students who write reading reactions based on structured questions would outperform significantly those 
students who only write a summary of an article. This was because the structured questions in this study are higher order 
cogmhve questions, which might stimulate the development of students’ cognitive strategies beyond simply memorizing content 
(Sanders, 1966). In other words, the structured questioning format requires the use of more metacognitive strategies and critical 
thinking skills. It requires not only analyzing skills but synthesizing, judging, and evaluating skills. As a result, the students who 
answer structured questions might have a better retention of the content while applying what they have read than those who only 
summarize the content. With the better retention, the structured questions group might do better on near transfer task. 

It was further expected that the structured questions group will perform significantly better on far transfer task than the 
summary group. This is based on the fact that structured questions in this study might activate the application of analyzing 
synthesizing and evaluating skills that are part of metacognitive strategies, and critical thinking skills. This is a kind of skill 
activation (Salomon, 1979). According to some researchers (King, 1992; Pressley, Wood, Woloshyn, Martin, King, & Menke, 
1992), generic questions (or structured questions in this study) are very effective at promoting critical thinking skills because they 
wil induce inference, evaluation, and verification. The writing of an EPSS (Electronic Performance Support Systems) product 
evaluation report, as a far transfer task, also involves applying synthesizing and evaluating skills. In contrast, although 
summarizing may also involve cognitive strategy activation, most people will focus on content learning when they summarize 
(Gamer, 1987). 
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Method 

Participants 

Sixty-six undergraduate students (7 males, 59 females) from a class of 101 students at Florida State University participated in 
the study as a part of an outside class assignment and a class activity. Although 101 students participated in transfer tasks (as a 
class activity), only 66 students submitted the outside class reading assignment. So the final result was based on the 66 students 
who participated in both activities in this study. Most students are sophomores or juniors who major in or plan to major in 
elementary, special, and early childhood education. All the students were enrolled in a course entitled Introduction to Educational 
Technology. Some students may know of the term CBT (Computer-Based Training), but they had never been exposed to the idea 
of EPSS (Electronic Performance Support Systems), the content topic of this study. The students’ participation in this study was 
assigned as part of their class activities. They did not get any extra credit points for participating, and their participation in this 
research study did not affect their grade or any negative consequences. 

Materials 

Two articles about Electronic Performance Support Systems (EPSS) were used for the reading assignment. EPSS is an 
interactive computer-based environment that provides on-the-job and just-in-time support to facilitate task performance and 
product development. It usually consists of a library and information support system, customized tools, learning function, and an 
interactive expert system. The first article is Leighton’s (1996) What is an EPSS ? It introduces the background, definitions, goal, 
and components of EPSS. The other is Sleight’s (1993) What is Electronic Performance Support and What Isn't? In the article, 
Sleight identifies major characteristics of EPSS and compares examples and non-examples of EPSS. Both articles are four pages 
long with about 1500 words. 

The EPSS product the students evaluated for the far transfer task is the Florida Curriculum Planning Tool (CPT) version 2.0. 
CPT was designed and developed by the Center for Performance Technology at Florida State University for the Florida 
Department of Education (DOE). The purpose of CPT is to facilitate teachers’ planning of learning activities and units of 
instruction that are in compliance with Florida Sunshine State Standards. It is designed, in particular, to assist teachers in 
developing learning activities for any subject, grade level, strand, theme, standard, benchmark or any combination of the above. It 
also allows teachers to import activities developed by other teachers, and export activities they develop to share with others. 

Independent Variable 

The independent variable in this study is the type of reading assignment, which had two formats. One format requires the 
student to write reading reactions based on the given structured questions. These questions include: 

• What are your gut reactions to the article? 

• What do you think are the big ideas in this article? 

• What are the implications for your (future) teaching or work? 

• What are the implications for your learning? 

• What questions does it leave unanswered? 

• What is your rating of the usefulness of this article? Choose one of the following: (poor, fair, good, excellent) 

The other format requires students to write a summary of the article in a free format. 

Dependent Variables 

The two dependent variables are near transfer and far transfer. The measurement of near and far transfer is the score on the 
writing of a list of EPSS evaluation criteria (near transfer) and the score on the CPT evaluation report (far transfer). The score of 
the EPSS evaluation criteria is determined by the number of evaluation standards that are included in the student’s evaluation 
based on the ideas from the two EPSS articles. The total possible score is 15, representing fifteen major points mentioned in the 
two articles about the characteristics of an EPSS. The score on the CPT evaluation report is determined by the following 
assessment criteria. These criteria are adopted and modified from the principle of authentic writing assessment (Chapman, 1990- 
Hart, 1994; Frederiksen & Collins, 1989). The authentic writing assessment examines a student’s writing holistically. One major 
characteristic of this method is that the students’ total scores consist of scores representing different dimensions which include 
content, organization, style, and mechanics. The formal writing assessment usually emphasizes writing style, mechanics of 
spe ling and syntax, and content. However, in this study, the focus was on the content. This is because the content in the final 
evaluation report should reveal how well they have learned from the reading assignment, and whether any transfer has occurred 
hour dimensions are used to score the evaluation report. They are listed below. 

1. Focus: the main idea, theme, or point of view is clear and consistently maintained, overall recommendation about the 
product included. (0-3) 

2. Support: arguments and conclusions are adequately supported and explained. (0-3) 

3. Organization: the logical flow of ideas is clear and related. (0-3) 

4. Content: Number of major points mentioned in the EPSS articles included. (0-6) 

On the basis of the above definitions, more detailed scoring principles were developed for each dimension. A three-point 
scale was used for the first three dimensions of Focus, Support, and Organization. A six-point scale was used for the dimension 
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of Content because the content was the key factor that was associated with the transfer of learning in this study. Thus, the total 
possible score for the far transfer task is 15 points. 

Procedure 

At the beginning of the course, the students were told about the study. The participants were randomly assigned to one of two 
groups: one was told to write reading reactions based on the structured questioning format, the other was told to write summaries 
of the articles in a free format. The experiment began in the third week of class. Both groups were asked to read two articles on 
the topic of EPSS, and then either answer structured questions or write a summary as an outside class assignment. They were 
required to submit their reading assignments at the beginning of the fourth week class. During the fourth week of class, the 
students were required to do two tasks in class, in an hour, without access to other materials. The first task was to write a list of 
an evaluation criterion for evaluating EPSS products based on what they had read. The second task was to write an evaluation 
report of the CPT. The students finished these two tasks in front of the computer in the computer lab. They were asked to browse 
the CPT first and write down their evaluation in essay format. The evaluation criteria and report were collected by the researcher 
after the class. 

The students’ evaluation reports were scored by the researcher and another doctoral student whose major is measurement 
and testing in the College of Education. Both raters have experience in teaching and writing assessment. The following procedure 
was used in scoring the students evaluation criteria and the evaluation report of CPT. Before starting the grading process, the 
raters read two EPSS articles and the exercise instructions to get familiar with the content and purpose of this study, which was 
very important for another rater. The researcher assigned the numbers one to sixty-six to participants for scoring purposes. 
Neither rater knew the student’s treatment group when grading his or her writing so as to avoid any bias to a specific group. 
Based on the scoring guide for grading, the two raters tried to score three evaluation reports together as a test. They then 
discussed the differences and came to a consensus for further grading. Finally, the two raters finished grading all sixty-six 
evaluation reports individually. To ensure the reliability and consistency between the two raters, correlation coefficients for both 
scores were calculated using the statistical software package, SPSS. The results are shown in Table 1.0. Pearson’s correlation 
coefficients for the two scores are .80 and .73. They are both statistically significant at the .01 level. 

After finishing the scoring, the researcher averaged the two raters’ given scores as the final score for each student. Two 
scores of each student on near and far transfer tasks were then input into the SPSS data worksheet for data analysis. 

Table 1.0 Correlations Between Two Raters on Transfer Task Scores 



Correlation 


Rater 


A 


B 


Evaluation Criteria 


A 


1.00 


0.80 




B 


0.80 


1.00 


Evaluation Report of CPT 


A 


1.00 


0.73 




B 


0.73 


1.00 



Note. Correlation is significant at 0.01 level. 

Results 

Relative Effectiveness on Near Transfer Task 

The score of each student’s performance on near transfer task was measured by the number of key points included in 
their list of evaluation criteria. The possible total score is 15, which represents 15 key points mentioned in the two EPSS articles. 
Table 2.0 presents the means, standard deviations, and rangp of scores for the students’ EPSS evaluation criteria. The average 
score of the structured questions group (M — 3.35) was lower than the average score of the summary group (M = 3.85), but the 
difference was not statistically significant, F(l, 64) - 2.257, g. > .05. 

Table 2.0 Mean Scores of Writing a List of Evaluation Criteria 



Measure 


Group 


N 


M 


SD 


Range 


Evaluation Criteria 


Structured Question 


34 


3.35 


1.00 


2.00-7.00 




Summary 


32 


3.86 


1.68 


1.00-9.00 




Total 


66 


3.60 


1.38 


1.00-9.00 



Note. The possible total score for evaluation criteria is 15 points. 



Relative Effectiveness on Far Transfer Task 

The student’s performance on the far transfer task was determined by their evaluation report of Curriculum Planning Tool 
(CPT). The evaluation report was in essay format. The scoring criteria included four dimensions, focus (3 points), support (3 
points), organization (3 points), and content (6 points) for a total possible score of 15 points. The means, standard deviations, and 
range of scores are displayed in table 3.0. Again, the summary group outperformed the structured question group. The average 
score of the summary group (M = 9.23) is a little higher than the average score of the structured question group (M = 8.94). 
However, the difference is not statistically significant. The result of a one-way ANOVA showed F(l, 64) = .19, £ = .664. With 
g>.05, the null hypothesis could not be rejected. 



Table 3.0 Mean Scores of the Evaluation Report of CPT 



Measure 


Group 


N 


M 


SD 


Range 


Evaluation Criteria 


Structured Question 


34 


8.94 


2.69 


3.00-14.50 




Summary 


32 


9.23 


2.77 


5.00-15.00 




Total 


66 


9.08 


2.71 


3.00-15.00 



Discussion 

The discussion will focus on the factors that might have affected the transfer tasks as a whole, since the possible reasons for a 
lack of statistical significance should be similar for both the near transfer task and the far transfer task. 

Although there were no previous studies that directly measured the effects of structured questions on transfer tasks, many 
studies (Applegate, Quinn, & Applegate, 1994; Sanacore, 1984; Andre & Anderson, 1978; Kosonen & Winne, 1995; Nisbett, 
1993; Perins & Grotzer, 1997) did suggest that metacognitive skills and critical thinking skills facilitate better learning. The 
structured questions used in this study were supposed to activate metacognitive skills and critical thinking skills. Thus, 
theoretically, the forced use of such skills as analyzing, synthesizing and evaluating in answering structured questions should 
have promoted reading comprehension and produced better understanding in applying knowledge and skills obtained through 
reading articles. 

Contrary to what was expected by theoretical assumptions, the structured questions group did not perform any better than the 
summary group on both near transfer task and far transfer task. The lack of a significant effect may be due to a number of factors: 
failure to activate the students’metacognitive skills or critical thinking skills, the lack of student motivation to the experiment, 
inadequate practice for the treatment, and the unfamiliar topic area with a lot of technical jargon in the articles used for the 
current study. 

Failure to Activate Metacognitive Skills or Critical Thinking Skills 

There are several possibilities that the students failed to activate metacognitive skills or critical thinking skills. One 
possible explanation is that most students’ answers to structured questions were about the article rather than the reflective 
thinking on the content of the article. The purpose of using structured questions in the current study was to force the student to 
apply critical thinking skills such as analyzing, synthesizing and evaluating skills to reflect on the implications of the content of 
the articles in their future learning or teaching. However, while answering questions like “what are your gut reactions to this 
article?”, “what are the implications for your future teaching? and “what are the implications for your learning?”, most students 
failed to recall details of the content of EPSS such as its components and functions, but rather focused on their general feelings. 
Those general feelings were emotional, and many were negative. For example, several students expressed their concerns that the 
computer would replace humans because of EPSS, which was not what they wanted to happen. Others complained there were too 
many technical terms in the articles that were difficult for them to understand at the beginning. 

On the other hand, the summary group at least applied their own metacognitive strategies to summarize the main idea and its 
supporting ideas. The group’s sole focus on the content might have helped them recall the content of articles when they were 
required to write the list of evaluation criteria for EPSS product evaluation. 

Another possibility of the failure of skill activation is that the structured questions in this study might have been too difficult 
to these students. This was inferred from an analysis of the students’ answers to implication questions. Most of them centered 
around describing the content of the articles rather than the implications for practice. If the students did not understand what 
inferences are, or simply did not answer the question, it is unlikely the structured questions had the desired effect. 

Lack of Students’ Motivation to the Experiment 

Participation in reading assignment phase of this study was voluntary though the participation of transfer tasks was required. 
However, the participation of the study had no negative effect on students’ grade. The students did not get reward for performing 
or not performing on the tasks. That might be the reason why only around one third of 101 students submitted their reading 
assignments. The completion of the reading assignment of writing summaries or answering structured questions happened outside 
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classes. The time the students spent finishing the assignment and the efforts they made were unknown. However, by reading their 
answers to the questions and summaries, it was somewhat obvious that many students did not put much effort into either 
answering the structured questions or writing summaries. Most students answered the structured questions with only one or two 
sentences or even left one question unanswered. Similarly, the summaries of most students were short with one paragraph of 
about 130 words. This may result in low retention of the content and poor performance on near transfer task among students of 
both groups. 

Another inference is that the students did not spend the time to learn or perform, since participation did not affect their grade. 
Perhaps they did not perceive the task to be worth a lot of effort. This low effort could be observed in the time spent in 
completing the transfer tasks. The transfer tasks were designed to be completed in about 40 to 50 minutes, but most participants 
in this study finished the tasks in 30 minutes or less. The expected finishing time was based on the average time the students 
spent in two pilot tests. The first pilot test was conducted in the same course during the previous semester. The students who 
attended the pilot test had similar backgrounds and the average time they spent was about 40 minutes, ranging from 30 to 50 
minutes. In another pilot test done in a graduate class, the average time the students spent was about 40 minutes. 

Inadequate Practice of the Treatment 

The short time period for the experiment might not have provided the structured questions group with enough practice on 
answering structured questions. The current study asked students to read two articles on the same topic within a week. The 
students who participated in this study were not familiar with the structured questions format for reading. Nor did they get 
feedback as to the accuracy of their responses. Two trials may not be adequate for them to make use of this format. In previous 
applications of this format as used in Construe (Lebow, Wager, Marks, & Gilbert, 1996), students received feedback after their 
first application which led to better responses on the second application. As a result, perhaps, the treatment was not optimized. 

Unfamiliar Topic Area of Reading Material 

The participants in this study had no prior knowledge of the topic of EPSS, and they were still a bit confused after reading the 
articles. The two EPSS articles used in this study are conceptual and well written with simple and very concise language. 
However, perhaps because of their conciseness, they did not provide examples to explain where the EPSS products were being 
used, and what they looked like. This caused problems with understanding for some students. 

The limited prior knowledge of EPSS might have hindered the students’ comprehension of the articles and weakened their 
performance on the transfer tasks. Participants in this study were from various educational majors. The course was one of the 
required courses for educational major students that help them establish a foundation of knowledge in educational technology. 
Most participants were novice computer users. When the experiment was conducted, it was in the early weeks of the course. The 
students were just beginning to learn Microsoft Word, and email. They had not studied educational software evaluation which 
was a topic to be discussed later in the semester, and is related to EPSS product evaluation. This lack of basic knowledge about 
evaluation may have affected their performance on the transfer task. A large number of previous studies (Hamilton, 1997; 
Machiels -Bongerts, et al., 1995; Wilson & Cole, 1992; Reigeluth, 1983) have indicated that prior knowledge facilitates 
information processing and text recall. Without related prior knowledge, transfer might be difficult to occur. When means of both 
summary and structured questions group scores were examined, it is evident that the average score was very low. For the first 
task, the total average score was only 3.60, which was 24.0% of the possible total score. The total average score of 9.08 for the 
second transfer task was only 60.5% of the total points which would also be considered failing performance. One assumption of 
these low scores is that the acquisition of knowledge never took place. Without the retention of the learning content, learning 
transfer would never have been occurred. 

Suggestions for Future Research 

While the hypotheses were not supported, the current study provides sufficient evidence that further research should be 
pursued in several areas for a better understanding of the effect of structured questions on transfer tasks. These areas include: 

• How to monitor or control student’s efforts in answering structured questions 

• How to provide feedback on the adequacy of answers to structured questions 

• How to provide adequate practice to better the chance of transfer 

• How to assess content acquisition before determining performance on transfer tasks 

Monitor or Control Student’s Efforts in Answering Structured Questions 

This study did not record the time the students spent to answer the structured questions. However, the students’ short answers 
to the questions revealed that many of the students might not engage in thinking with metacogiitive skills or critical thinking 
skills that were expected to be activated. Without the activation of the skills, the transfer of learning barely occurred. To assure 
the occurrence of transfer, future research may include directions that set the minimum length of each answer to the questions, 
and require the answer to focus on the content. Another adjustment for the research can be to ask the students to finish the 
reading assignment in the class. In this way, both structured questions group and summary group will have the same amount of 
time to finish the assignment. The time each student spends on the reading can also be recorded and analyzed. 
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Provide Feedback on the Adequacy of Answers to Structured Questions 

One recommendation for future research is to provide feedback or a model to students for their answers to structured 
questions. The feedback should be directed toward getting appropriate and accurate answers to the questions related to key ideas 
and implications. The results of this study suggest that many students did not make use of the structured questions format well. 
To make sure the students answer the questions more adequately, it may be more appropriate and effective to provide each 
student feedback on their answers after the first reading, and show them some good examples of answers as models. The 
feedback may result in better application of the structured questions format and a better chance of activation of metacognitive 
skills and critical thinking skills. 

Extend the Duration of the Experiment 

It is also recommended that the duration of the experiment be extended. The results of this study indicate that students might 
not get used to answering the structured questions after reading only two articles. It might show significant difference if the 
students are required to read two more articles for two more weeks rather than to read two articles within a week. The increased 
amount of practice with the structured questions format may increase the possibility of activating metacognitive skills and critical 
thinking skills that the students have acquired before. The skill activation may in turn generate better transfer. 



Assess Content Acquisition Before Determining Performance on Transfer Tasks 

It is recommended that future researchers assess content acquisition before conducting transfer tasks. A test of recall may be 
conducted after students finish reading articles to examine whether students acquire knowledge or not. 

Future research may also select another topic other than EPSS. It is recommended that participants not be familiar with the 
chosen topic but are interested in it because they will put in more effort and participate in the activities more actively if they 
perceive something as relevant and interesting (Keller, 1983). It is also recommended that the transfer task be designed so that it 
can be assessed properly, if not easily. 

In conclusion, this study shows that under the stated conditions both structured questions and summarizing strategies are 
equally effective. However, it is likely that different conditions would lead to different results. That is, modified conditions in 
future research such as training on the use of structured questions, a meaningful and relevant task, and motivated participants 
might change the outcomes of this study. 
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